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In the coming era of exascale supercomputing, in-situ visualization is an inevitable approach to 
reduce the output data size. A problem of the in-situ visualization is that it loses interactivity 
unless a steering method is adopted. In this paper, a new method is proposed for interactive 
analysis of in-situ visualization images produced by a batch simulation job. A key idea is to apply 
a lot of — from thousands to millions of — in-situ visualizations at once. Then the viewer analyze the 
^ ' image database interactively in the post processing. When each movie is compressed to the order of 

, 10 MB, the total size of one million movies is only the order of 10 TB that is smaller than the size 

of raw numerical data in exascale supercomputing. A feasibility study of the proposed method is 
performed. Multiple movie files are produced by a simulation and they are analyzed with a specially 
designed movie player. One can interactively change the view angle, visualization method, and their 
parameters by retrieving a proper sequence of images form the movie data set. 
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I. INTRODUCTION 



In supercomputer simulations, post processing is becoming the bottleneck in the whole research cycles. This is 
because the disk I/O and the network bandwidths do not catch up with the exponential growth of the computer 
processor speed. To resolve this imbalance, one has to apply some kind of data reduction to the output data in 
supercomputer simulations. One approach is to directly compress the raw numerical data. Discrete Fourier transforms 
and wavelet transforms are often used for that purpose. Another approach to the output data reduction is to apply 
. ! the visualization at run-time. This approach, called in-situ visualization, is effective to reduce the data size since 
■ images are spatially two-dimensional. 

The in-situ visualization poses new challenges to visualization researchers. New difficulties arise from the fact that 
today's supercomputers are massively parallel machines. The target data to be visualized are divided into pieces that 
are distributed in network-connected memories. New visualization methods and algorithms should be implemented 
on supercomputers for the in-situ visualization. They have to have a good parallel scaling in order not to degrade the 
simulation's scale. These challenges make a focus of attention in the HPC (High Performance Computing) community 
on the in-situ visualization. Recent study on the in-situ visualization for HPC are reviewed in section iHIl 

A key point in the in-situ visualization is proper settings of two kinds of parameters; visualization parameters and 
l/^ \ camera parameters. The visualization parameters include applied visualization methods and their states and values. 
I" ■ The camera parameters include the viewpoint position, direction, field of view, and viewing frustum culling. 
I ' A simple in-situ visualization with one fixed camera, without steering, is conceptually described in Fig. QJi, in 
which a 3-dimensional space spanned by three bases of the camera parameters, the visualization parameters, and the 
simulation time. A white ball in this figure denotes a visualization shot taken in the 3-D space. After the simulation 
task is finished, visualized images are sent to a local disk drive of the data analyzer's PC. The white arrow in the 
figure indicates the data transfer. As a result of this in-situ visualization, the sequential images are shown as a movie 
\ on the PC's window. 

In most simulations, key phenomena or events usually appear only in localized, discrete spots in the whole simulation 
region. If this kind of a "hot-spot" is beside the range of the camera view, one has to re-set the camera positions in 
the so-called scenario file and re-submit a simulation job, unless the steering simulation approach is taken. 

In the steering simulation, the analyzer can dynamically control both the simulation and the visualization parameters 
in the run-time. A steering simulation with in-situ visualization is symbolically described by Fig[TJ). The visualized 
images are transferred to the analyzer at run-time of the simulation; see the white arrow in Fig. QJ>. Watching the 
images on the PC monitor, the analyzer can dynamically change the visualization parameters to focus on a hot 
spot. The dashed line with arrow in Fig. [TJj stands for the feedback control from the analyzer to the simulation and 
visualization. Although this kind of the steering simulation is certainly an effective approach for HPC, it is not always 
possible because supercomputer simulations are usually performed in batch jobs. 
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FIG. 1: Concepts of various visualization approaches, (a) Post processing. The camera-axis denotes the freedom of the viewing 
such as the viewpoint positions and angles. The visualization-axis denotes visualization methods and their internal parameters. 
The third axis is the simulation time. A ball in this camera-visualization-time space stands for a visualization shot, (b) Steering 
simulation and visualization, (c) Bullet-time method, (d) Steering visualization of in-situ visualization images, proposed in 
this paper. 

Without the steering, the in-situ visualization completely loses interactivity, which is critically important to get 
insights from the simulation. 

The purpose of this paper is to propose a method that enables an interactive viewing of visualization results 
generated by in-situ visualization of a batch-job supercomputer simulation. 

II. INTERACTIVE VIEW OF IN-SITU VISUALIZATION DATA 

The key idea of the method proposed in this paper is to apply a cluster of in-situ visualizations with different 
visualization and camera parameters at once and analyze the output image data interactively as a post processing. 

The proposed method can be regarded, in some sense, as a generalization of the so-called bullet-time method used 
in the film industry, in which cameras are placed along a specified path and pictures are taken sequentially with short 
time intervals. It is used to make a slow motion movie of the target object that is apparently taken by a camera moving 
with impractically high speed. Fig. [TJ; shows a conceptual image of the bullet-time method in which a visualization 
is taken from four different cameras. As in the case of Fig. QJ,, the output data is a sequence of images that is sent to 
the analyzer afterward. 

In our method, we apply as many in-situ visualization as possible and analyze the output image data later in an 
interactive way. See Fig. [T}i. Any information on the simulated phenomena is supposed to be in the data set of the 
images, and the analyzer can dynamically explore the image space. 

When an output movie from one in-situ visualization is compressed to a reasonable size, say 10 MB, the total output 
data size is only 10 TB even the number of applied in-situ visualization is one million. This data is still smaller than 
the size of raw numerical data in exascale simulations that would amount to the order of PB. 
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When we pick up only the camera position among the camera parameters, our proposed method imply that we 
place a lot of cameras inside and outside the simulation region and apply the in-situ visualization in parallel for all 
of them; see Fig. [2] This configuration of cameras would remind us the "3D Dome" or "3D Room" constructed by 
Kanade et al. for capturing a human motion in a room-sized space to construct a virtualized reality 0, Q • Our method 
could be regarded as an application of the 3D room concept to the scientific visualization. An important difference is 
that our in-situ visualization method, cameras can be placed inside of the target without affecting it. 



The output data of our proposed method is a data set of lots (thousands to millions) of movie files. When the 
camera number is enough, we can find practically any image from any position in the data set. Therefore, one can 
"rotate" a visualized object by the mouse motion, by extracting a proper sequence of image files from the movie data 
set. 

To demonstrate the feasibility of the proposed method, we have performed a batch simulation on a PC cluster 
system in which 130 cameras are placed around the simulation region. Three kinds of in-situ visualization methods 
are applied from all of the cameras at the run time. The output data is 130 x 3 = 390 sets of MPEG movie files. We 
have developed an interactive movie player by which a sequence of images is extracted from the data set. 



The in-situ visualization has a long history. Refer [3] for a general review of steering simulation and visualization. 
Examples of in-situ visualization of peta-scale simulation, and their technical challenges (especially those caused by 
massively parallel processing), are summarized in 0, @- They show that the in-situ visualization is a promising 
solution for the peta- and exa-scale simulation. A natural extension of the online dynamical control of simulation 
is the end-to-end approach [f| 0], in which even the mesh generation, which is usually done in the preprocess stage, 
is done on supercomputers. They developed a steering simulation for seismic wave propagations whose visualization 
images were shown in real time. 

Huang et al. have developed a steering simulation and visualization framework for the environmental science by 
which dynamical control through the Internet is realized [8|. Their user interface is constructed on the web browsers. 
Ellsworth et al. have developed an in-situ visualization system on the Colombia supercomputer for a weather forecast 
model Q . (Our experimental system described in section PVTl is close to their system in the sense that multiple MPEG 
files are generated by multiple in-situ visualizations. While their movies are shown separately in each panel of a tiled 
display system, ours are used as a database.) Esnard et al. have developed an in-situ visualization system in which 
parallel visualization processing can run on different computer system from the simulation computer system [Io| . 
Special emphasis in their development was placed on making a steering environment with existing simulation codes. 

A lot of in-situ visualizations, with or without the steering, were designed and developed for particular simulation 
problems. On the other hand, general visualization frameworks with high parallel scalability are becoming available 
recently. Whitlock et al. [ll[ have developed a library Libsim that enables in-situ visualizations using Visit, which 
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FIG. 2: A cluster of visualization cameras for in-situ visualizations. 



III. RELATED WORK 
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is one of the most sophisticated parallel visualization tools available today. Their paper has also a concise review 
of the history and the latest status of the in-situ visualization study. Fabian et al. [12j report development of 
a coproccessing library for ParaView, another sophisticated parallel visualization tool. With that library, one can 
utilize various visualization functions provided by ParaView at run-time, being decoupled from the simulation. Latest 
case studies of in-situ visualization with Visit and Para View can be found in [13j . 



IV. CONFIGURATION OF VIEWPOINTS 

One of key points in our approach to the interactive in-situ visualization is the configuration of multiple cameras. In 
the extreme case, cameras will be placed in a 3-dimensional distribution as dense as possible inside as well as around 
the simulation region. In the feasibility test of the present paper, we place only 130 cameras with two-dimensional 
distribution on a spherical surface of a fixed radius from the center of the simulation region, to keep the test being in 
a moderate level. All the cameras look at the center. 




FIG. 3: A spherical camera distribution on the latitude-longitude grid viewed from the due north. 

Since there are only five regular polyhedra, it is not a straightforward task to place more than 20 cameras on a 
spherical surface as uniform as possible. 

Placing cameras on each grid point of the spherical coordinate system, or the so-called latitude-longitude mesh, does 
not work well because it leads to concentrations of cameras around the poles. Reducing camera numbers in higher 
latitudes is not a good idea because a camera motion along a constant longitude will meet a lacked viewpoint (Fig. [3]) . 
To avoid this problem, we make use of Yin- Yang grid [1J] in which two congruent component grids, which are a part 
of low latitude region of the latitude-longitude mesh, are combined with partial overlap to cover a spherical surface 
(Fig.©. 

V. INTERACTIVE VIEW OF MANY MOVIES 

The camera position is not the only parameter that can be explored in our method. See the parameter axis in 
Fig. [TJi. Different visualization methods and their internal parameters such as isosurface level, slice position, color 
functions etc. can also be explored in our method. An ultimate goal of our method is to take very many of — hopefully 
thousands of — different visualization parameters from each camera. But here in this test, we take only 3 different 
visualization parameters. Details will be described in the next section. The total number of output movie file is, 
therefore, 390 (= 130 cameras x 3). 

Those movie files that contain a lot of information would be useless unless a proper player. A movie player for PCs 
usually does, basically, a simple task; it reads one source movie file and shows a sequence of images contained in the 
source file the screen. Here we need a generalized player that reads multiple (390 in the present test and millions in 
future) source files, and displays a sequence of images from them. The play should be interactive: The source should 
be instantly and smoothly changed by a user's input event such as the mouse motion or key type-in. 



FIG. 4: Cameras (yellow balls) placed on a spherical surface around the simulation region (light blue box) used in the experiment 
described in section IVII To avoid concentrations of cameras near the poles, Yin- Yang grid configuration is adopted, (a) 65 
cameras are placed on the Yin-grid, which is a low latitude part of the spherical polar coordinates, (b) Another set of 65 
cameras are placed on the Yang grid which has the same shape as Yang-grid but with rotated configuration, (c) The Yin- and 
Yang grids are combined to cover a whole spherical surface. 




FIG. 5: A special movie player developed in this study. Multiple movie files are loaded into the player and a sequence of images 
from different movie files are retrieved and shown on the screen. 

We have developed such an interactive movie player with OpenCV library. A snapshot is shown in Fig. [5] It can 
retrieve a sequence of images from different movie files and display them on the window as a movie. The player we 
have developed has three functions; (i) to stop and play of the motion, (ii) to change of the source movie, and (iii) to 
zoom-in/out of a scene. 

The function of zoom-in/out is realized by the image magnification provided by OpenCV. If we placed many other 
cameras in a fully three-dimensional distribution, we would be able to perform the zoom-in/out function by chaining 
the source cameras. 

VI. APPLICATION TO SEISMIC WAVE PROPAGATION SIMULATION 

As a feasibility study, we have performed a seismic wave propagation simulation with the proposed in-situ visual- 
ization method. The simulation and visualization codes were developed by Furumura and Chen. Since the details of 
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the codes are described in their paper [15j , here we briefly summarize key featured relating to our in-situ visualization 
experiment. 

In this simulation, the time development of seismic waves in Cartesian coordinates (x, y, z) is solved. Basic variables 
are the displacement field u(x,y,z) of a viscoelastic medium and its velocity v(x, y, z). The viscoelastic equations of 
motion are solved for them with a high-order finite difference method. A pseudo-spectral method can also be chosen 
in their code when the spatial accuracy is critically important. We apply only the finite difference method in this test. 

One-dimensional domain decomposition in the z (vertical) axis is applied for the parallel simulation. The rectangular 
simulation region is divided into horizontal slates. This code has a good, almost linear, scaling up to more than 1000 
processors of the Earth Simulator supercomputer 15]. The hybrid parallelization with MPI and OpenMP is adopted. 

Furumura and Chen have also developed visualization tools for this seismic wave propagation simulation. These 
tools are derived from a highly optimized parallel visualization package [l6[ developed in GeoFEM framework [TtJ for 
the Earth Simulator supercomputer. The original visualization package is designed to adopt unstructured cell data. 
Here in this visualization tool for the seismic wave propagation, they are applied to the Cartesian structured mesh. 
Various visualization methods based on parallel ray castings are implemented in the original package for GeoFEM. 
In this test, we apply only one kind of visualization method, i.e., the volume rendering for scalar fields. 

Seismic waves have two modes; s-wave (shear mode) and p-wave (compressional mode). Defining two scalar fields 
4> s = |V x v| (for s-wave) and <f> p = V • v (for p-wave), we have applied the in-situ volume rendering visualization to 
the two fields 4> p and <j) s . 

We have performed test runs on a PC cluster system. The simulation grid size is 256 x 512 x 160. Number of 
MPI processes is 40. As we mentioned above, 130 cameras are spherically placed on the Yin- Yang grid points — 
65 for Yin and 65 for Yang. Each camera performs three visualization tasks; volume renderings for the s-wave, 
for the p-wave, and for both of them. Visualization snapshots are taken for 60 times within one simulation job. 
The images are saved in the PPM format, with the size of 2.5 MB per image. The total size of all the images is 
57 GB (= 2.5 MB x 130 cameras x 3 tasks x 60 frames). After the simulation job, each set of sequential images for 
one camera is combined into a movie file in the MPEG format by the FFmpeg library. (We will implement run-time 
compression in future.) Compression rate is about 3 %: The PPM image set of 57 GB in total was converted into 
MPEG files of 1.7 GB in total. 

If we would save the raw numerical data of 4> s and <p p with the single precision for the post processing, it would 
amount to 10 GB (= 256 x 512 x 160 grids x 4 B x 2 x 60 snapshots). Our movie data (1.7 GB) is an order of 
magnitude smaller than that. This gap will increase much more in larger scale simulations. 

The total number of movie files produced from this experiment is 390 (= 130 cameras x 3 tasks). The 390 MPEG 
files of 1.7 GB are transferred to a PC shown in Fig. [5] and they are loaded to the interactive movie player described 
in Section [V] 

Fig. [5] shows a sequence of snapshots of interactive viewing of the in-situ visualization movie data. Propagation of 
p-wave (orange) and s-wave (green) are observed in the movie on a PC window. Typing a key on the keyboard send 
a signal to the movie player to change the source movie file (or to change the camera angle). One can observe the 
phenomena from any one of the spherically distributed 130 cameras shown in Fig. 21 The image retrieving from the 
movie data set is smooth and fast. One can change the camera position while the seismic wave is propagating in the 
window. In spite of the relatively small number of cameras, the view change is so smooth that one may misunderstand 
that this is the usual post-process visualization. 

Fig. [7] shows snapshots of an interactive switch of visualization tasks. In the beginning, both the p-wave and s-wave 
are visualized in the window. At this moment, the movie player is retrieving an image sequence from a movie file 
that contains images of both the p- and s-waves. When the viewer types a key, the movie player changes the source 
file and starts retrieving image sequence from the new source that contains, for example, only the s-wave images. 
The window now shows the s-wave propagation. The user does not feel any delay in this switch. Another key typing 
causes again an instant change of the visualization task to the p-wave propagation. 

Fig. [8] shows the zooming function of the player. As we have mentioned in section [V] we will place dense cameras 
in the 3-D space as uniform as possible in the ultimate implementation in future. In that case, if we want to focus on 
a specific small region in the simulation region, our interactive movie player will extract other image sequence taken 
from a camera located closer to the target region. In the current test, on the other hand, we have implemented the 
zooming by the magnification of images taken from a fixed camera. 

VII. CONCLUDING REMARKS 

In this paper, we have proposed a new in-situ visualization method for exascale simulations. This method applies 
a lot of (hopefully millions of) in-situ visualizations at once with (thousands of) different visualizations taken from 
(thousands of) different cameras. The output in this strategy is not numbers, but movies. 
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Even when a simulation produces one million of movies, the total output data size is only 10 TB when each movie 
is compressed to 10 MB. It should be noted that the size of 10 TB is rather small in the coming exascale era. If one 
tries to save raw numerical data with the same temporal resolution for the standard post process visualization, the 
output data would amount to much more. 

The output of million movies will be analyzed as a post processing in our proposed method. A specially designed 
movie player will read the million movie files and display a sequence of images in a window. By getting a proper image 
sequence from different movie files, we can effectively "rotate" the visualization objects while the dynamic phenomena 
are shown in the window. We can also change the adopted visualization method with different internal parameters 
as long as the corresponding in-situ visualizations have been applied in the simulation job. 

The user will feel that the simulation is running at that moment. The experience will be the same as the steering 
simulation. 

To demonstrate the feasibility of the proposed method, we have performed a simulation in which 390 in-situ 
visualizations are applied. We have also developed a movie player that reads the 390 movie files and play a movie on 
a PC window. With this player, it is confirmed that we can interactively "rotate" a visualized object, and can change 
the visualization tasks in real time by the key typing while the movie is playing. 

Our proposed method can be summarized as the following steps: (i) Identify potential ranges of the visualization 
and the camera parameters in which the user will interested; (ii) Discretize the parameter subspace as fine as possible; 
(iii) Apply in-situ visualizations to all the discrete parameters (white balls in Fig. [T]i); and save the output movies; 
and finally (iv) Explore the movie data space by an interactive movie player. Any information necessary for the 
analyzer will be in the movie data set. 

Our experiment has led us to conclude that this method — in-situ visualization with interactive view — is not only 
feasible, but also practical for peta- and exa-scale supercomputing. 
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FIG. 6: A snapshot sequence of the movie player. One can rotate the view angle by typing the arrow keys (right/left/up/down) 
on the keyboard, while the p-wave (orange) and s-wave (green) are propagating. Note that this is not a steering simulation or 
a post process visualization in the usual sense. The simulation has been already done by a batch-job. 
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FIG. 7: Interactive control of other visualization parameters than the camera position. One can change the visualization task, 
volume renderings of p-wave (right panel), s-wave (left panel), and both of them (left panel), by typing a key in the keyboard, 
while the wave propagation is showing. 




FIG. 8: Close-up view of the data. The zooming is implemented by an image processing. In this case, the magnification rates 
are of 1.0 (left panel), 1.5 (middle), and 2.0 (right). When other cameras are placed in closer to the target, one can change the 
source to the camera as in the case of the camera rotation shown in Fig. [6] 



